On 20100816 at 06:26 AM Julian Junge typed...
> option a.) When a mail is send to a closed ticket, the mail bounces, and the sender receives a notification that his case is closed and he please removes the ticket number from the subject and sends that mail again.

You could create a reply directing the user what he/she needs to do.  In this case ADMIN | QUEUE | Select Queue then click CHANGE button | Modify Follow-Up Option to reject.

Create/Modify appropriate Auto-Queue response to respond with the appropriate verbiage.



On 20100816 at 06:26 AM Julian Junge typed...
>option b.) The mail does not bounce but  a new case is created and OTRS replaces the ticket number in the subject.

Configure the Follow-up option for the queue - ADMIN | QUEUE | Select Queue then click CHANGE button | Modify Follow-Up Option to New Ticket

Create/Modify appropriate Auto-Response<-->Queue responder to respond with a message indicating the ticket has been opened under a new ticket.



I think there are default auto-responses available when you initially configure these options, but customizing the auto-responder is IMO a highly recommended step.