The Meta-owned app is testing video selfies with facial analysis software as a new age-verification method.
Some users on Instagram try to skirt its 13+ age rule by editing their date of birth to make them appear over 18.
But US teens attempting this will now be given three ways to verify age: upload ID, ask three adult users to vouch for them or take a video selfie.
Meta says it hopes the new methods will ensure teens have an "age-appropriate experience" on Instagram.
The tech giant has previously faced criticism over teen and child safety on its platforms.
Several US states probed Instagram last year over children's experiences on the photo-sharing app, in response to leaks from Facebook whistleblower Frances Haugen.
Will Gardner OBE, chief executive of Childnet and director of the UK Safer Internet Centre, says the trial is encouraging: "The potential is there to try and help protect children from content which isn't for them and make their internet experience more age-appropriate."
5Rights Foundation, a UK organisation campaigning for child safety in the digital environment, says such efforts are "long overdue".
Platforms should "leave behind the 'don't look don't see attitude' that has led to millions of children being put at risk", 5Rights says, adding that "simply knowing the age of your users is not enough."
Parents and guardians of teen Instagram users were given additional tools to supervise their child's experience on Instagram earlier this month.
They can now set up time limits and view details of any reports their child makes on the platform.
Teens will also be "nudged" to look at other content if repeatedly viewing the same topics on Instagram's explore page, and encouraged to "take a break" if continuously scrolling through reels.
Video selfies and social vouching
Video selfies have become a popular way for digital platforms - such as online banking apps - to verify users' age or identity.
Instagram currently uses video selfies as one way account holders can verify their identity if they get locked out of their account.
META
Meta has partnered with UK digital identification provider Yoti, whose technology estimates age by analysing human faces and facial features.
Yoti says its algorithm, trained on anonymous people's facial images and their date of birth, cannot individually identify users or anything about them, except their age.
Its latest white paper, published in May, said the technology was accurate for six to 12-year-olds with an error range of 1.36 years - and an error range of 1.52 years for 13 to 19-year-olds.
Meta says both companies will delete the image once a user's age has been confirmed.
Meanwhile, social vouching allows users to ask three mutual followers to confirm their age. Those being asked to confirm how old a user is must be at least 18, and cannot be vouching for any other users at the time.
Feeling safe on social media
Dr Ysabel Gerrard, lecturer in digital media and society at the University of Sheffield, says Instagram's new age-verification methods are a welcome addition to just asking users to upload ID.
But she says relying on age-verification tools as a way to protect young people online can overlook why they actually try to create adult accounts in the first place.
"A lot of them are saying they're 18 on Instagram not to do bad things or view bad content," Dr Gerrard says.
"Being technically registered as an adult makes them feel safe because they don't think they're going to be targeted."
For Dr Gerrard, Instagram's new verification mechanisms raise larger questions about what helps children to feel safe on social media platforms.
"Pretending to be an adult is one of them. It's a harsh reality, but we can't pretend it's not happening."