Yes, Instagram can detect screen recording in some cases. For example, if the story creator has enabled the 'Hide Story from' feature and includes those who might screen record, Instagram may flag it. Also, there are certain analytics tools available to creators that can give them an idea if their content is being screen recorded more often than normal, though it's not a 100% accurate detection method.
Instagram doesn't directly notify the story creator that you specifically are screen recording. However, there are some indirect signs. For instance, if a large number of people start screen recording a particular story, it might show up in the overall analytics data of the creator. But from a regular user's perspective, just one person screen recording is not something that can be easily spotted by another user.