Thursday, April 14, 2011

Taking Tests

I wrote this after reading a thread at RevitForum called Autodesk Certification.

In the thread there, a quick mock-up "test" that one member submitted is a good example (I think) of why it's hard to make a test. When we write a question for a test, it and the answers we offer have a bias. If five people submitted their own 15 question tests, I'm confident that anyone who takes it would be able to take exception with at least a couple questions in each. That's why test instructions say "pick the BEST answer", which usually translates into, the one that the test's writer wants selected.

Oddly enough, I often think that people with a lot of experience can have a rougher time with a test because they are aware of exceptions to the rule, all the ways that something can be twisted into being. That and these tests are written by people that are presumed to have significant experience, but what kind of experience? training?

How about this question and set of answers? (which btw I've actually seen, not word for word but the essence of that test's question)

In-Place Families...
  • a) should never be used.
  • b) should only be used sparingly.
  • c) can only used for custom casework.
  • d) are best used for sloped walls.
  • e) are used for massing.

  • Answer A is an opinion
  • Answer B is a recommendation
  • Answer C suggests an office preference or standard
  • Answer D is too specific/limiting and suggest a preference
  • Answer E is true, people often forget that massing is in-place families.

They are all "true" depending on your point of view. Which one did I intend to be the "best" answer? Perhaps "E"? It depends on what I wanted to test "for". Revit knowledge or appreciation of subtleties? Applying Revit to our office standards?

    A test's wrong answers can be as interesting as the correct ones. Failing the "right" kind of test might actually demonstrate deeper knowledge of the subject.

Something Aaron Maller once wrote somewhere (AUGI/RevitForum/Blog) that his employer (Beck Group) strives to hire the best architect/engineer they can. They'll teach them the software if necessary. It's a good strategy assuming "you" aren't hunting for an office guru. If nobody in your office has guru status it will be a bit harder to select one (evaluate one), he/she only has to know more than your smartest user to "impress".

There is a lot more a potential candidate can do to prove their value besides showing they passed a test. Remember getting a license to practice architecture (no small feat) means that person passed the minimum qualifications to be able to do so. These other things could be their past work history, references, teaching experience, problem solving examples, writing, and speaking. Oh and the interview shouldn't just be with the HR folks, how will they know they've got the right person? With something like this?

HR Check List:
  • Nice suit/outfit - check
  • Grooming Excellent - check
  • Revit tattoo on forehead - check
  • Degree at BIM Tech - check
  • Autodesk Certification - check
  • Can spell Rivit - check
Recommendation: Hire at 3x the salary

Imagine being eliminated from consideration only because you didn't take the Autodesk test? At best a test is just one of many possible ways to evaluate someone's fitness for a job.

As for testing and an unsolicited plug, you could check out Knowledge Smart.

I got a chance to take one of their beta tests last year or so, been awhile. Their process involves using Revit and provides a nice summary of results afterward. It isn't inexpensive, but they'd like to remind us that the cost of hiring the "wrong" person can be pretty costly. So their testing methodology isn't pennywise/pound foolish. Fellow blogger David Light recently posted a recommendation after talking about them with Robert Manna (another blogger). I vaguely recall hearing the conversation while we were at Autodesk last week.

As for the original question at RevitForum, does anyone recommend taking the Autodesk certification exam? The answer remains elusive and personal. Why do you need it? Is it a requirement? Do you like taking tests and having certificates? Will it help differentiate you from other people at your company, your region, your realm of influence? If many people take it will it no longer be special or rare? In the job hunt, are you competing for a position with just a few others or hundreds? These days with so many out of work it might well be the latter. A candidate will have to have a resume that has some pretty special stuff in it...regardless of Revit certification.

Is there an answer in there somewhere? Only you can know for sure...good luck!

[edit: Seth Godin posted something this morning that I thought was slightly related.]


Aaron Maller said...

It *is* worth mentioning that what i posted in that thread was NOT my idea of "testing questions," they are follow up questions from training days, which- i would wager- is very different.

If i were asking questions, they wouldnt be multiple choice.

I dont think its a bad thing that answers to firms tests are firm specific or test writer specific, as in your example. I think thats why it is paramount the tests be write in, or "make something" based. That way a wrong answer can tell a lot, versus a multiple choice.

As for the guru thing... Thats also why i was saying i think, at that point, its about making phone calls and getting references. In the spirit of "you dont know what you dont know," if youre not looking to hire staff- but a guru, how can you possibly test them on something you dont know?

I dont know squat about doing taxes, but i managed to find someone to do mine... With references. :)

ShawnF said...

"Qualified" and "Certified" are two totally different things, and neither are necessarily related.