What do you consider the "cost" of the certification process?

For the sake of providing a concrete example let's consider 360 submission. I first think of the cost of certification as the time between shipping the game and being "done" with the game. The time I don't get to spend finishing the game. The polish time I don't get to have. The time I need to set aside for getting the team ready to...wait.

I suspect many people don't realize that from the developer's perspective, the game has to be done long before you send it off to manufacturing. It has to be done several weeks before that, in fact, so that (in this example) Microsoft can verify it's bug-free, compliant with technical requirements, etc.

This period of time requires you to have a significant number of people available for the inevitable top-priority bugs that MS comes back with. They may be rare crash bugs that you can't reproduce on your own (but of course the QA guys at MS can repro it enough to fail your submission). They may be multiplayer bugs that you never see on your internal network. They may be ninja controller pulls or crazy PC system specs problems or UI localization or... point being, the bugs that come back during submission or rarely simple to fix but are obviously the highest imaginable priority. I mean, if you don't fix these the game won't ship. So you've got to have folks on hand to fix this stuff during the submission process, even though everyone's pretty much sitting back with a "we did it, it's over" feeling.

When those bugs come back from your 1st submission, you have to have people on hand that can fix them, correctly and quickly. Those people aren't taking time off to recharge before the next project, they aren't doing pre-production, they're just...not doing work. And you have to budget for two submissions, so that's twice as much not doing work. Now, since you budget for two submissions you can assume you're going to fail your first sub (which is a very reasonably plan...MS can be pretty demanding) and therefore you keep fixing bugs and polishing the game. Then, when the Fail comes back from MS you take care of the bugs they found and your second submission is actually a better version of the game.

This actually bit us in the keister on Xmen2, though, where we had at least one bad bug that wouldn't prevent the game from being certified but we did want to fix before the game went to consumers. So we sent Sony the first submission thinking "we'll be able to take the next few days to fix this issue, then the first submission will fail and we'll include this fix in our second submission." Unbelievably, our first submission passed. We couldn't believe it. We wound up petitioning Sony to allow a second submission that included the fix for Bug X. Let me tell you one way to get Sony's hackles up...actually pass their submission process and then ask to resubmit anyway. Well, you, the consumer, did get a better version of the game out of it, at least.

So, yeah. That's the cost of certification to me. A number of talented people sitting around not making the game better. Maybe a good question for the future would be, "Is the cost of certification worth what you get out of it?"