I spoke about this to some extent at NSConference 2009. There are really two things that are going to differ depending on whether it's an in-house component or for release: default behaviour, and flexibility/configurability.

My stance is that any UI component should be designed such that it's not coupled to the project you created it for, and that it should at least display _something_ by default. Use the existing AppKit classes as a guide, and maintain loose coupling with delegate protocols, well-designed/named API methods and notifications where appropriate. Remember to always pass the control itself as a parameter to delegate methods (so the receiver knows _which_ control sent the message), and consider whether it might be useful to do so in the userinfo dictionary for your notifications. Prefer mimicking APIs that other developers will already be familiar with (such as standard data-source protocols like "tell me how many, then give me each one in turn", or the "should/will/did" permission and notification pattern).

The default behaviour for an in-house component can be exactly you need it to be, but the default behaviour for a released piece of code should be whatever seems most generally useful or applicable. Put some serious thought into the average use case. If someone sees your control and decides to download it, what are they expecting it to do? Does it actually do that thing? If it's reasonable to assume that your control does a particular thing by default, make sure it actually does do that thing.

Regarding flexibility and configuration, it's important to strike a balance between limitations and the spaghetti code that comes from unlimited configurability. Let common sense be your guide. Try to keep in mind that if you design your API correctly, and factor your code into appropriate methods, anyone wanting to do serious customisation can subclass instead of calling dozens of configuration setters. Decide what options will serve 70% or so of the usage situations you can think of, and provide those options. Let your API design serve the rest via the ability to override methods in (as thin as possible) subclasses.

As a final note, if you're doing a lot of custom UI work, keep a sticky note someplace to remind you to consider the big 4 commonly-forgotten factors: Sizes (most controls come in three sizes; yours should too), Variants (your control might benefit from standard variants, like rounded textfields), Resolution (try to make your drawing code resolution-independent), and Accessibility (it's easy to provide a basic level of accessibility support, and it makes a huge difference to a lot of people).