Moving an item in your inventory results in the item losing its CustomModelData nbt tag if it has the AttributeModifiers nbt tag.
Tested using the following:
give @p minecraft:iron_axe{CustomModelData:1b,AttributeModifiers:[{AttributeName:"generic.attack_damage",Slot:"mainhand",Amount:8,Operation:0,UUID:[I;1,0,0,1]},{AttributeName:"generic.attack_speed",Slot:"mainhand",Amount:-3.1,Operation:0,UUID:[I;2,0,0,2]}]} 1give @p minecraft:iron_axe{CustomModelData:1b,AttributeModifiers:[{AttributeName:"generic.attack_damage",Slot:"mainhand",Amount:8,Operation:0,UUID:[I;1,0,0,1]},{AttributeName:"generic.attack_speed",Slot:"mainhand",Amount:-3.1,Operation:0,UUID:[I;2,0,0,2]}]} 1Using the next command it shows the data getting removed after you move it in your inventory.
data get entity @s Inventory[0].tag.CustomModelDatadata get entity @s Inventory[0].tag.CustomModelData
After posting I realised the server I was on was on a modified version. After testing in single player it does not occur.
May still happen on unmodified servers but I am unable to test that.