Mattis told reporters that "Russian elements" moved across a deconfliction line into an area on the eastern side of the Euphrates river where the sides had previously agreed they could operate, he said.
But he said they came "too close" to positions of US soldiers in the area.
The chairman of the US Joint Chiefs of Staff, General Joe Dunford, contacted his Russian counterpart, General Valery Gerasimov, about the incursion.
"And those elements fell back so we have also drawn off slightly to maintain the deconfliction between the elements there," he said.
"So it seems that this time, it was resolved through the deconfliction communication line," he added.
Mattis did not give a precise date for the incident, but on Thursday the Pentagon issued a brief statement noting that Dunford and Gerasimov had held talks on Syria and other matters of mutual interest.
According to a US official, the Russian mercenaries have been moving into an area near Deir Ezzor in eastern Syria .
It was when they began digging firing positions that the US troops deployed in the area along side their Syrian Democratic Force allies became concerned, said the official, who spoke on condition of anonymity.
The incident comes less than two months after an attack in the same area on the headquarters of the SDF, an alliance of US-backed Kurdish and Arab fighters.
US soldiers were in the building and alerted the US-led coalition, which responded with an air strike that killed 100 to 200 fighters, including numerous Russians.
Evoking that attack, Mattis noted that it was led by "Russian mercenaries." It was the first time that he directly accused Russia of involvement. APP/AFP