I don’t think there is much doubt that there is a gender-related issue going on with office hours — and, in fact, most responsibilities that require face-to-face student time. The Chronicle of Higher Education has documented this at some length. This could be because of the actual interests and personalities of the faculty, but also seems clearly to be related to the demands of students and colleagues. Minority faculty also experience this.
So what can you do? Here is what I do: at the beginning of the semester, when I’m covering course policies, I explain what the office hour requirements are. Here is literally what I say:
“Part of my job here at the university is teaching classes and mentoring students. This includes time preparing for classes and grading. Part of my job is helping with the university process by being on committees and helping with university project. Another important part is contributing to the historical profession by keeping up with reading, doing research, attending conferences. The university wants me to do these things and is paying me to do them. In order to facilitate the part of my job that is teaching, and which I enjoy very much, the university requires all faculty to have ____ office hours. It is sometimes possible that I or my colleagues will be in the office more than that, but we are encouraged to make sure the rest of our time is spent on the other tasks the university pays us for. So, here are my office hours. I will make sure to be there at these times unless I tell you otherwise (because of being gone for conferences or illness or other). I will have a sign-up sheet on my door for these office hours. It is possible that in rare situations I will agree to meet with you outside these hours, but if I do, you should understand that I am making a special effort and going out of my way — not doing other things because I am doing this. You should know this is true for all your faculty. You are expected to meet with us in our stated office hour time. If I or another faculty agree to meet with you at another time, you really want to make sure you show up — you don’t know how we have rearranged things to make that possible.”
This is exactly what I say. I get really good results with this. I think students are interested in knowing what faculty do. They are really ignorant of it. I also meet with lots of students outside the stated office hours (especially if it is something I have required of them, like conferencing over a paper, and the students have classes during my scheduled time).
And then you have to just ignore the student evaluations on this subject. If you do indeed have regular hours that you keep and that meet the stated requirements of the university, any outside evaluator will just assess that and realize the students are being demanding. It is hard to have these complaints be so unfairly targeted by gender, but I bet you’ll find most of your colleagues also feel students are over-demanding of them, too, in some way and will be sympathetic.
As to your gender question — I hadn’t thought about it, but I wouldn’t be at all surprised if you were right. I haven’t compared notes with my male colleagues, but I can tell you that the women at my university are the doers, and they’re always more available. While there is probably little progress we can make with our students (and perhaps ourselves!) in regard to gender-skewed expectations, I always find that setting clear boundaries from the get-go is the healthiest approach. At my university, there is actually a policy requiring a certain number of office hours. Pass the buck! “Per university policy, I will be in my office and available to you ____ each week. Meanwhile, I can answer emails within ___ hours.” Sometimes dealing with students is like a study in natural selection: our desire to please them actually signals our weakness to them, and they’re always ready to throw us under the bus. So my advice is to take up a tough demeanor when it comes to your own time. Make it clear that you have grading, research, and service commitments on top of face-to-face time with them. Tell them that you hold office hours as sacred — you will always be there and available to talk — but you also hold your other commitments outside of those hours as sacred as well. Not only will this set the correct "don’t-mess-with-me" tone, but it will also be a great example to them of how mature people handle being busy without being harried . . . especially for your female students!
Although I can’t give you a citation for a study, I hear it said widely that students expect women faculty to be more accessible (as well as more supportive and nurturing in a wide variety of ways), and also take it for granted when this is the case. So it’s possible that the comments on your being less available primarily come from this issue.
Another thing that could be going on is that your office hours are less-well-matched to student schedules than some of your other colleagues. One way to judge this is to compare how heavily attended your office hours are, compared to those of some of your colleagues teaching similar courses. (This can be complicated because attendance is also affected by how likely the student thinks the instructor is to be helpful, how difficult the course is, etc., but try to pick a good comparison.)
I schedule only three hours of office hours per week, because I can’t schedule all the other things I need to if I have too many office hours. I tell my students on the first day of class that I really value meeting with them outside of class and hope they take advantage of the opportunity; however, I have few office hours scheduled because there is no way for me to schedule times that will work for everyone, and if they want to see me and can’t attend my scheduled office hours, they should contact me by email or speak to me after class to make an appointment. If I’m teaching a large introductory class, I add that they will get the most out of such an appointment if they come with a study partner or two, and prepare for the meeting by identifying together what questions they have for me. (If I’m teaching a small upper level course for physics majors, I don’t necessarily say this, although I do encourage them to work together.)
In addition, students find when they come to ask me questions that I always start by asking them to explain what they already understand, and then ask a series of questions aimed at helping them figure out how to answer their own question. So they have to work pretty hard in office hours! I have found that this approach seems to work pretty well and I spend a modest but not inordinate amount of time meeting with students outside of class.
I also think carefully about what time to schedule my office hours. The best times are very specific to a particular campus so what I do may not be best in your setting. For most of my career, I have scheduled 1.5-2 hours per week of evening office hours and the rest daytime. I teach at a small residential liberal arts college and most students are free in the evening. I started this when I had small children and I would leave work around 2:30 or 3 in the afternoon, go home, spend the late afternoon and dinner time with them, put them to bed, and go back to work at about 8. Students loved it because it was easy to come to my office hours. I loved it because I was too tired for anything but office hours at that point, and this kept my daytime hours free for other work and for family. However, in the last 2-3 years I have not done this, as now I have teenage children who need to be taken to activities in the evening, and who are not necessarily home in the afternoon. So I’ve tried to find other times of day when most students are free.