Can not delete \bin\roslyn\VBCSCompiler.exe - Access is denied

I am facing a strange issue from roslyn compiler. Sometimes when I build the solution I face a strange issue in error list which does not let me build the solution. Here is the error:

Severity Code Description Project File Line Suppression State Error Unable to copy file "D:\Sealogical New Website\SealogicalWebsite\packages\Microsoft.Net.Compilers.1.0.0\tools\csc.exe" to "bin\roslyn\csc.exe". Access to the path 'bin\roslyn\csc.exe' is denied. SealogicalWebsite Severity Code Description Project File Line Suppression State Error Unable to copy file "D:\Sealogical New Website\SealogicalWebsite\packages\Microsoft.Net.Compilers.1.0.0\tools\VBCSCompiler.exe" to "bin\roslyn\VBCSCompiler.exe". Access to the path 'bin\roslyn\VBCSCompiler.exe' is denied. SealogicalWebsite

I try to clean the solution and rebuild it but does not work again. Tried to delete files manually but again shows an error pointing out I do not have right permissions to do this: Tried deleting from CMD but again shows the same error:

https://i.stack.imgur.com/XUcwF.png

The only way to delete roslyn generated files is by restarting the PC and when restarted I can delete them manually without a problem.

I am administrator in my PC so there is no problem with that I guess.

This problem seems to happen in Visual Studi0 2015 and Visual Studio 2017.

I dont like restarting the PC 4-5 times a day or more thats why I am asking here.

Some other details:

I am the one who created this project so theoretically it should not be a problem to do this action.

When the project first created I did run the Visual Studion in Administrator Mode as I am doing now.

I am using ASP.MVC 5 template in my project.

Files which do not get deleted are bin/roslyn/csc and bin/roslyn/VBCSCompiler.

I could find a solution to remove this part from web.config so it does not generate those files:

<compilers>
  <compiler language="c#;cs;csharp" extension=".cs" type="Microsoft.CodeDom.Providers.DotNetCompilerPlatform.CSharpCodeProvider, Microsoft.CodeDom.Providers.DotNetCompilerPlatform, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" warningLevel="4" compilerOptions="/langversion:6 /nowarn:1659;1699;1701" />
  <compiler language="vb;vbs;visualbasic;vbscript" extension=".vb" type="Microsoft.CodeDom.Providers.DotNetCompilerPlatform.VBCodeProvider, Microsoft.CodeDom.Providers.DotNetCompilerPlatform, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" warningLevel="4" compilerOptions="/langversion:14 /nowarn:41008 /define:_MYTYPE=\&quot;Web\&quot; /optionInfer+" />
</compilers>

But this solution would cause problem with new c# 6 features so I can not remove this section.....

Furthermore I tried this command in CMD to see if there is a process locking this file but anyway it did not show me any process at all:

C:\WINDOWS\system32>tasklist /m D:\SealogicalNewWebsite\SealogicalWebsite\SealogicalWebsite\bin\roslyn\csc.exe
INFO: No tasks are running which match the specified criteria.

Does anybody knows a way how to handle this strange issue ?