And he suggested some colleagues who do not agree with his approach on Brexit were trying to undermine him.
Minister Liam Fox said he "deplored" the briefing by some of his colleagues.
The international trade secretary told the BBC's Sunday Politics they should "be very quiet" and "stick to their own departmental duties", adding: "Our backbenchers are furious and the only people smiling at this will be in Berlin and Paris."
Since the general election, cabinet splits have surfaced over the issue of the 1% cap on public sector pay rises, with some ministers pressing for it to be lifted.
Labour is promising £4bn which it says would offer a pay rise to workers.
On the BBC's Andrew Marr Show, Mr Hammond defended his stance, saying public sector pay had "raced ahead" of the private sector after the economic crash in 2008.
While in terms of salary alone, that gap had now closed, he continued, when "very generous" pension contributions were taken into account, the 10% disparity between public and private salaries was a "simple fact".
"I do think on many fronts it would be helpful if my colleagues - all of us - focused on the job at hand," he said.
He added: "If you want my opinion, some of the noise is generated by people who are not happy with the agenda that I have, over the last few weeks, tried to advance, of ensuring that we achieve a Brexit which is focused on protecting our economy, protecting our jobs and making sure that we can have continued rising living standards in the future."
Mr Fox, one of the leading Brexit campaigners in the cabinet, rejected press reports he had clashed with Mr Hammond over the EU, saying the two had a "very good working relationship".
"I don't know where the briefing is coming from, but I do know it's got to stop," he said.
He added: "I think there's too much self-indulgence, and I think people need to have less prosecco and have a longer summer holiday."
Former Tory leader Iain Duncan Smith criticised those briefing against Prime Minister Theresa May, saying: "Just for once shut up, for God's sake, and let everybody else get on with the business of governing."
Pay gap
Pay rises for most public sector workers are set by independent pay review bodies, but have effectively been capped at 1% each year since 2013.
Before that, there was a two-year freeze on pay for all but the lowest-paid workers.
The government has come under pressure over the policy since the general election, with some Conservative ministers speaking out in favour of lifting the cap.
Shadow chancellor John McDonnell said Labour would spend £4bn on ending the cap, insisting this would be enough to give a real-terms increase for public sector workers.
Pay review bodies would be asked to come up with an "honest judgement" and a Labour government would follow their advice, he said.
On Pienaar's Politics on BBC Radio 5 live, First Secretary of State Damian Green was asked whether Mr Hammond said public sector workers were "overpaid".
"I'm not going to report from inside cabinet because cabinet ministers should not do that," he said.
"But the chancellor does not think that public sector workers are overpaid - the government obviously respects the millions of people who do really important jobs."
