Her current appointment ends in November after this year's event realised record ticket sales and capacity attendances.
"I had the option to re-apply for the job, but decided that it was timely to let someone else take the reins," Ms Caffin said.
"The festival and the Writers' Centre are in such a strong position that now is the opportune time for someone with a new vision to take over. That's what keeps festivals alive and growing."
Even though she is stepping down as director, Ms Caffin will continue with the festival as an adviser and consultant.
Northern Rivers Writers' Centre secretary Russell Eldridge said it was wonderful to have Ms Caffin back for the 2013 festival.
"She is an extraordinary woman and festival director," he said.
Mr Eldridge said a new festival director should be in place by early November.
"Jeni has already started the planning for the 2014 festival so we expect the handover to be seamless," he said.
Ms Caffin was publicist for the event in 2005 and 2006 and then director from 2007-10 before taking over again as director for this year's event.
"My highlights this year were the presentations by Peter Carey and DBC Pierre's conversation with festival chairman Chris Hanley," she said.
"It was one of the best literary conversations I had ever heard.
"And of course the Bedroom Philosopher pole dancing at the end of the festival was a perfect Byron festival moment for me."
Update your news preferences and get the latest news delivered to your inbox.