Wowza Community

Catching the FileUtils.copyFile2 : java.io.FileNotFoundException

Hi everyone. I have an issue with the movement of recorded files to S3. Sometimes, actually quite rarely, I catch the following exceptions:

ERROR	server	comment	2016-03-18	05:40:47	-	-	-	-	-	117788.092	-	-	-	-	-	-	-	-	FileUtils.copyFile2 : java.io.FileNotFoundException: /mnt/s3/camera_4_6fa289c4-cfba-4581-ab5b-82b15dcbe48c.stream_2016_03_18_053945375.mp4 (No such file or directory)|at java.io.FileOutputStream.open0(Native Method)|at java.io.FileOutputStream.open(FileOutputStream.java:270)|at java.io.FileOutputStream.<init>(FileOutputStream.java:213)|at java.io.FileOutputStream.<init>(FileOutputStream.java:162)|at com.wowza.util.FileUtils.copyFile2(FileUtils.java:153)|

sometimes it’s a single exception as in the example above, and sometimes it appears together with other exceptions as in the example below

ERROR	server	comment	2016-03-24	12:18:34	-	-	-	-	-	83973.669	-	-	-	-	-	-	-	-	Cannot execute query: java.sql.SQLException: Could not retrieve transation read-only status server|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:973)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:949)|
ERROR	server	comment	2016-03-24	12:18:34	-	-	-	-	-	83973.672	-	-	-	-	-	-	-	-	[live/_definst_]LiveStreamRecorderMP4: wowza35_6844f9d6-f798-45b1-b42b-276ce02ee672.stream: /usr/local/WowzaStreamingEngine-4.3.0/content/wowza35_6844f9d6-f798-45b1-b42b-276ce02ee672.stream_2016_03_24_121733829.mp4: java.lang.RuntimeException: Cannot execute query|at com.l1inc.wowzacustomize.RecorderWriteListener.onSegmentEnd(RecorderWriteListener.java:73)|at com.wowza.wms.livestreamrecord.manager.StreamRecorder.notifySegmentEnd(StreamRecorder.java:754)|at com.wowza.wms.livestreamrecord.manager.StreamRecorder$LiveStreamRecordActionListener.onSegmentEnd(StreamRecorder.java:58)|at com.wowza.wms.livestreamrecord.model.LiveStreamRecorderBase.notifySegmentEnd(LiveStreamRecorderBase.java:568)|at com.wowza.wms.livestreamrecord.model.LiveStreamRecorderMP4.b(LiveStreamRecorderMP4.java:138)|
ERROR	server	comment	2016-03-24	12:18:34	-	-	-	-	-	83973.678	-	-	-	-	-	-	-	-	Cannot execute query: java.sql.SQLException: Could not retrieve transation read-only status server|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1084)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:987)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:973)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:918)|at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:949)|
ERROR	server	comment	2016-03-24	12:18:36	-	-	-	-	-	83975.554	-	-	-	-	-	-	-	-	FileUtils.copyFile2 : java.io.FileNotFoundException: /usr/local/WowzaStreamingEngine-4.3.0/content/wowza35_6844f9d6-f798-45b1-b42b-276ce02ee672.stream_2016_03_24_121733829.mp4 (No such file or directory)|at java.io.FileInputStream.open0(Native Method)|at java.io.FileInputStream.open(FileInputStream.java:195)|at java.io.FileInputStream.<init>(FileInputStream.java:138)|at com.wowza.util.FileUtils.copyFile2(FileUtils.java:152)|at com.wowza.wms.module.ModuleMediaWriterFileMover$CopyWorker.run(ModuleMediaWriterFileMover.java:84)|

As you can see I use WowzaStreamingEngine-4.3.0.

I use several additional modules:

  • com.wowza.wms.module.ModuleMediaWriterFileMover to move files to the mounted S3 bucket.

  • custom FileNameDelegate to rename recorded files

  • custom RecorderWriteListener to capture information about each recorded segment and to save this information in a database.

Here are examples of these modules:

public class FileNameDelegate extends ModuleBase implements IStreamRecorderFileVersionDelegate {
    static DateFormat dateFormat = new SimpleDateFormat("yyyy_MM_dd_HHmmssSSS");
    public String getFilename(IStreamRecorder recorder) {
        String name;
        Date date = new Date();
        try {
            File file = new File(recorder.getBaseFilePath());
            String oldBasePath = file.getParent();
            String oldName = file.getName();
            String oldExt = "";
            int oldExtIndex = oldName.lastIndexOf(".");
            if (oldExtIndex >= 0) {
                oldExt = oldName.substring(oldExtIndex);
                oldName = oldName.substring(0, oldExtIndex);
            }
            if ((file = new File(name = String.valueOf(oldBasePath) + "/" + oldName + "_" + dateFormat.format(date) + oldExt)).exists()) {
                file.delete();
            }
        }
        catch (Exception e) {
            getLogger().error("MyDelegate.getFilename: " + e.toString());
            name = "temp" + dateFormat.format(date) + ".tmp";
            throw new RuntimeException(e); 
        }
        return name;
    }
}

public class RecorderWriteListener extends ModuleBase implements IStreamRecorderActionNotify {
	private static final String DB_URL = "dnaDBurl";
	private static final String DB_USER = "dnaDBuser";
	private static final String DB_PWD = "dnaDBpwd";
	private Long startRecordTime = null;
	private Long endRecordTime = null;
	private static ComboPooledDataSource cpds;
	private static final String INSERT_QUERY = "INSERT INTO dnamain.CameraRecordFile (filename, id_camera, starttime, endtime, duration, filesize) " +
			"SELECT ?, c.id, ?, ?, ?, ? FROM dnamain.Camera c WHERE c.streamname = ?";
	
	public void onAppStart(IApplicationInstance appInstance) {
		initDBConnection(appInstance);
	}
	
	public void onAppStop(IApplicationInstance appInstance) {
		destroyDBConnection();
	}
	
	@Override
	public void onSegmentEnd(IStreamRecorder recorder) {
		if (startRecordTime == null)
			startRecordTime = recorder.getStartTime().getMillis();
		
		String fileName = recorder.getCurrentFile().replaceAll("/usr/local/WowzaStreamingEngine-4.3.0/content/", "");
		String streamName = recorder.getStreamName().replaceAll(".stream", "");
		Timestamp startTime = new Timestamp(startRecordTime);
		Long duration = recorder.getCurrentDuration();
		
		endRecordTime = startRecordTime + duration;
		
		Timestamp endTime = new Timestamp(endRecordTime);
		Long fileSize = recorder.getCurrentSize();
		startRecordTime = endRecordTime;
		try (Connection connection = cpds.getConnection()) {
			try (PreparedStatement insertStatement = connection.prepareStatement(INSERT_QUERY)) {
				try {
					insertStatement.setString(1, fileName);
					insertStatement.setTimestamp(2, startTime);
					insertStatement.setTimestamp(3, endTime);
					insertStatement.setLong(4, duration);
					insertStatement.setLong(5, fileSize);
					insertStatement.setString(6, streamName);
				} catch (SQLException e1) {
					getLogger().error("Cannot set parameter", e1);
					throw new RuntimeException("Cannot set parameter", e1);
				}
				
				try {
					insertStatement.executeUpdate();
				} catch (SQLException e) {
					getLogger().error("Cannot execute query", e);	
					throw new RuntimeException("Cannot execute query", e);
				}
			} catch (SQLException e) {
				getLogger().error("cannot prepare statement", e);
				throw new RuntimeException(e); 
			}
		} catch (SQLException e) {
			getLogger().error("cannot open connection", e);
			throw new RuntimeException(e); 
		}		
	}
	@Override
	public void onCreateRecorder(IStreamRecorder recorder) { 
	}
	
	@Override
	public void onSegmentStart(IStreamRecorder recorder) {
	}
	
	@Override
	public void onSplitRecorder(IStreamRecorder recorder) {
	}
	
	@Override
	public void onStartRecorder(IStreamRecorder recorder) {
	}
	
	@Override
	public void onStopRecorder(IStreamRecorder recorder) {
	}
	
	@Override
	public void onSwitchRecorder(IStreamRecorder recorder, IMediaStream newStream) {
	}
	
	private void initDBConnection(IApplicationInstance appInstance) {
		WMSProperties wmfp = appInstance.getProperties();
	
		String dbURL = wmfp.getPropertyStr(DB_URL);
		String dbUser = wmfp.getPropertyStr(DB_USER);
		String dbPwd = wmfp.getPropertyStr(DB_PWD);
		
		if (cpds == null) {
			cpds = new ComboPooledDataSource();
		}
		
		try {
			cpds.setDriverClass("com.mysql.jdbc.Driver");
		} catch (PropertyVetoException e) {
			getLogger().error("Cannot load JDBC driver", e);
			throw new RuntimeException(e);
		}     
		cpds.setJdbcUrl(dbURL);
		cpds.setUser(dbUser);                                  
		cpds.setPassword(dbPwd);
		cpds.setDebugUnreturnedConnectionStackTraces(true);
		cpds.setUnreturnedConnectionTimeout(10);
		
	}
	
	private void destroyDBConnection() {
		if (cpds != null) {
			cpds.close();
			cpds = null;
		}
	}
}

And live/Application.xml example

<?xml version="1.0" encoding="UTF-8"?>
<Root version="1">
	<Application>
		<Name>live</Name>
		<AppType>Live</AppType>
		<Description></Description>
		<!-- Uncomment to set application level timeout values
		<ApplicationTimeout>60000</ApplicationTimeout>
		<PingTimeout>12000</PingTimeout>
		<ValidationFrequency>8000</ValidationFrequency>
		<MaximumPendingWriteBytes>0</MaximumPendingWriteBytes>
		<MaximumSetBufferTime>60000</MaximumSetBufferTime>
		<MaximumStorageDirDepth>25</MaximumStorageDirDepth>
		-->
		<Connections>
			<AutoAccept>true</AutoAccept>
			<AllowDomains></AllowDomains>
		</Connections>
		<!--
			StorageDir path variables
			
			${com.wowza.wms.AppHome} - Application home directory
			${com.wowza.wms.ConfigHome} - Configuration home directory
			${com.wowza.wms.context.VHost} - Virtual host name
			${com.wowza.wms.context.VHostConfigHome} - Virtual host config directory
			${com.wowza.wms.context.Application} - Application name
			${com.wowza.wms.context.ApplicationInstance} - Application instance name
			
		-->
		<Streams>
			<StreamType>live</StreamType>
			<StorageDir>${com.wowza.wms.context.VHostConfigHome}/content</StorageDir>
			<KeyDir>${com.wowza.wms.context.VHostConfigHome}/keys</KeyDir>
			<!-- LiveStreamPacketizers (separate with commas): cupertinostreamingpacketizer, smoothstreamingpacketizer, sanjosestreamingpacketizer, mpegdashstreamingpacketizer, cupertinostreamingrepeater, smoothstreamingrepeater, sanjosestreamingrepeater, mpegdashstreamingrepeater, dvrstreamingpacketizer, dvrstreamingrepeater -->
			<LiveStreamPacketizers>cupertinostreamingpacketizer, mpegdashstreamingpacketizer</LiveStreamPacketizers>
			<!-- Properties defined here will override any properties defined in conf/Streams.xml for any streams types loaded by this application -->
			<Properties>
			</Properties>
		</Streams>
		<Transcoder>
			<!-- To turn on transcoder set to: transcoder -->
			<LiveStreamTranscoder></LiveStreamTranscoder>
			<!-- [templatename].xml or ${SourceStreamName}.xml -->
			<Templates>${SourceStreamName}.xml,transrate.xml</Templates>
			<ProfileDir>${com.wowza.wms.context.VHostConfigHome}/transcoder/profiles</ProfileDir>
			<TemplateDir>${com.wowza.wms.context.VHostConfigHome}/transcoder/templates</TemplateDir>
			<Properties>
			</Properties>
		</Transcoder>
		<DVR>
			<!-- As a single server or as an origin, use dvrstreamingpacketizer in LiveStreamPacketizers above -->
			<!-- Or, in an origin-edge configuration, edges use dvrstreamingrepeater in LiveStreamPacketizers above -->
			<!-- As an origin, also add dvrchunkstreaming to HTTPStreamers below -->
			<!-- If this is a dvrstreamingrepeater, define Application/Repeater/OriginURL to point back to the origin -->
			<!-- To turn on DVR recording set Recorders to dvrrecorder.  This works with dvrstreamingpacketizer  -->
			<Recorders></Recorders>
			<!-- As a single server or as an origin, set the Store to dvrfilestorage-->
			<!-- edges should have this empty -->
			<Store></Store>
			<!--  Window Duration is length of live DVR window in seconds.  0 means the window is never trimmed. -->
			<WindowDuration>0</WindowDuration>
			<!-- Storage Directory is top level location where dvr is stored.  e.g. c:/temp/dvr -->
			<StorageDir>${com.wowza.wms.context.VHostConfigHome}/dvr</StorageDir>
			<!-- valid ArchiveStrategy values are append, version, delete -->
			<ArchiveStrategy>append</ArchiveStrategy>
			<!-- Properties for DVR -->
			<Properties>
			</Properties>
		</DVR>
		<TimedText>
			<!-- VOD caption providers (separate with commas): vodcaptionprovidermp4_3gpp, vodcaptionproviderttml, vodcaptionproviderwebvtt,  vodcaptionprovidersrt, vodcaptionproviderscc -->
			<VODTimedTextProviders></VODTimedTextProviders>
			<!-- Properties for TimedText -->
			<Properties>
			</Properties>
		</TimedText>
		<!-- HTTPStreamers (separate with commas): cupertinostreaming, smoothstreaming, sanjosestreaming, mpegdashstreaming, dvrchunkstreaming -->
		<HTTPStreamers>cupertinostreaming, mpegdashstreaming</HTTPStreamers>
		<MediaCache>
			<MediaCacheSourceList></MediaCacheSourceList>
		</MediaCache>
		<SharedObjects>
			<StorageDir>${com.wowza.wms.context.VHostConfigHome}/applications/${com.wowza.wms.context.Application}/sharedobjects/${com.wowza.wms.context.ApplicationInstance}</StorageDir>
		</SharedObjects>
		<Client>
			<IdleFrequency>-1</IdleFrequency>
			<Access>
				<StreamReadAccess></StreamReadAccess>
				<StreamWriteAccess>*</StreamWriteAccess>
				<StreamAudioSampleAccess></StreamAudioSampleAccess>
				<StreamVideoSampleAccess></StreamVideoSampleAccess>
				<SharedObjectReadAccess>*</SharedObjectReadAccess>
				<SharedObjectWriteAccess>*</SharedObjectWriteAccess>
			</Access>
		</Client>
		<RTP>
			<!-- RTP/Authentication/[type]Methods defined in Authentication.xml. Default setup includes; none, basic, digest -->
			<Authentication>
				<PublishMethod>digest</PublishMethod>
				<PlayMethod>block</PlayMethod>
			</Authentication>
			<!-- RTP/AVSyncMethod. Valid values are: senderreport, systemclock, rtptimecode -->
			<AVSyncMethod>senderreport</AVSyncMethod>
			<MaxRTCPWaitTime>12000</MaxRTCPWaitTime>
			<IdleFrequency>75</IdleFrequency>
			<RTSPSessionTimeout>90000</RTSPSessionTimeout>
			<RTSPMaximumPendingWriteBytes>0</RTSPMaximumPendingWriteBytes>
			<RTSPBindIpAddress></RTSPBindIpAddress>
			<RTSPConnectionIpAddress>0.0.0.0</RTSPConnectionIpAddress>
			<RTSPOriginIpAddress>127.0.0.1</RTSPOriginIpAddress>
			<IncomingDatagramPortRanges>*</IncomingDatagramPortRanges>
			<!-- Properties defined here will override any properties defined in conf/RTP.xml for any depacketizers loaded by this application -->
			<Properties>
			</Properties>
		</RTP>
		<MediaCaster>
			<RTP>
				<RTSP>
					<!-- udp, interleave -->
					<RTPTransportMode>interleave</RTPTransportMode>
				</RTSP>
			</RTP>
			<StreamValidator>
				<Enable>true</Enable>
				<ResetNameGroups>true</ResetNameGroups>
				<StreamStartTimeout>20000</StreamStartTimeout>
				<StreamTimeout>20000</StreamTimeout>
				<VideoStartTimeout>0</VideoStartTimeout>
				<VideoTimeout>0</VideoTimeout>
				<AudioStartTimeout>0</AudioStartTimeout>
				<AudioTimeout>0</AudioTimeout>
				<VideoTCToleranceEnable>false</VideoTCToleranceEnable>
				<VideoTCPosTolerance>3000</VideoTCPosTolerance>
				<VideoTCNegTolerance>-500</VideoTCNegTolerance>
				<AudioTCToleranceEnable>false</AudioTCToleranceEnable>
				<AudioTCPosTolerance>3000</AudioTCPosTolerance>
				<AudioTCNegTolerance>-500</AudioTCNegTolerance>
				<DataTCToleranceEnable>false</DataTCToleranceEnable>
				<DataTCPosTolerance>3000</DataTCPosTolerance>
				<DataTCNegTolerance>-500</DataTCNegTolerance>
				<AVSyncToleranceEnable>false</AVSyncToleranceEnable>
				<AVSyncTolerance>1500</AVSyncTolerance>
				<DebugLog>false</DebugLog>
			</StreamValidator>
			<!-- Properties defined here will override any properties defined in conf/MediaCasters.xml for any MediaCasters loaded by this applications -->
			<Properties>
			</Properties>
		</MediaCaster>
		<MediaReader>
			<!-- Properties defined here will override any properties defined in conf/MediaReaders.xml for any MediaReaders loaded by this applications -->
			<Properties>
			</Properties>
		</MediaReader>
		<MediaWriter>
			<!-- Properties defined here will override any properties defined in conf/MediaWriter.xml for any MediaWriter loaded by this applications -->
			<Properties>
			</Properties>
		</MediaWriter>
		<LiveStreamPacketizer>
			<!-- Properties defined here will override any properties defined in conf/LiveStreamPacketizers.xml for any LiveStreamPacketizers loaded by this applications -->
			<Properties>
			</Properties>
		</LiveStreamPacketizer>
		<HTTPStreamer>
			<!-- Properties defined here will override any properties defined in conf/HTTPStreamers.xml for any HTTPStreamer loaded by this applications -->
			<Properties>
			</Properties>
		</HTTPStreamer>
		<Manager>
			<!-- Properties defined are used by the Manager -->
			<Properties>
			</Properties>
		</Manager>
		<Repeater>
			<OriginURL></OriginURL>
			<QueryString><![CDATA[]]></QueryString>
		</Repeater>
		<StreamRecorder>
			<Properties>
				<Property>
					<Name>streamRecorderFileVersionDelegate</Name>
					<Value>com.l1inc.wowzacustomize.FileNameDelegate</Value>
					<Type>String</Type>
				</Property>
				<Property>
					<Name>streamRecorderNotifyListener</Name>
					<Value>com.l1inc.wowzacustomize.RecorderWriteListener</Value>
					<Type>String</Type>
				</Property>
				<Property>
					<Name>streamRecorderDebugEnable</Name>
					<Value>true</Value>
					<Type>Boolean</Type>
				</Property>
				<Property>
					<Name>streamRecorderStartOnKeyFrame</Name>
					<Value>true</Value>
					<Type>Boolean</Type>
				</Property>
				<Property>
					<Name>streamRecorderMoveFirstVideoFrameToZero</Name>
					<Value>true</Value>
					<Type>Boolean</Type>
				</Property>
				<Property>
					<Name>streamRecorderSplitOnTcDiscontinuity</Name>
					<Value>true</Value>
					<Type>Boolean</Type>
				</Property>
				<Property>
					<Name>streamRecorderVersioningOption</Name>
					<Value>version</Value>
					<Type>String</Type>
				</Property>
			</Properties>
		</StreamRecorder>
		<Modules>
			<Module>
				<Name>base</Name>
				<Description>Base</Description>
				<Class>com.wowza.wms.module.ModuleCore</Class>
			</Module>
			<Module>
				<Name>logging</Name>
				<Description>Client Logging</Description>
				<Class>com.wowza.wms.module.ModuleClientLogging</Class>
			</Module>
			<Module>
				<Name>flvplayback</Name>
				<Description>FLVPlayback</Description>
				<Class>com.wowza.wms.module.ModuleFLVPlayback</Class>
			</Module>
			<Module>
				<Name>ModuleCoreSecurity</Name>
				<Description>Core Security Module for Applications</Description>
				<Class>com.wowza.wms.security.ModuleCoreSecurity</Class>
			</Module>
			<Module>
				<Name>RecorderWriteListener</Name>
				<Description>RecorderWriteListener</Description>
				<Class>com.l1inc.wowzacustomize.RecorderWriteListener</Class>
			</Module>
			<Module>
				<Name>FileNameDelegate</Name>
				<Description>FileNameDelegate</Description>
				<Class>com.l1inc.wowzacustomize.FileNameDelegate</Class>
			</Module>
			<Module>
				<Name>AutoRecordStreams</Name>
				<Description>AutoRecordStreams</Description>
				<Class>com.l1inc.wowzacustomize.AutoRecordStreams</Class>
			</Module>
			<Module>
				<Name>ModuleMediaWriterFileMover</Name>
				<Description>ModuleMediaWriterFileMover</Description>
				<Class>com.wowza.wms.module.ModuleMediaWriterFileMover</Class>
			</Module>
		</Modules>
		<!-- Properties defined here will be added to the IApplication.getProperties() and IApplicationInstance.getProperties() collections -->
		<Properties>
			<Property>
				<Name>securityPublishRequirePassword</Name>
				<Value>true</Value>
				<Type>Boolean</Type>
			</Property>
			<Property>
				<Name>fileMoverDestinationPath</Name>
				<Value>/mnt/s3</Value>
			</Property>
			<Property>
				<Name>fileMoverDeleteOriginal</Name>
				<Value>true</Value>
				<Type>Boolean</Type>
			</Property>
			<Property>
				<Name>fileMoverVersionFile</Name>
				<Value>true</Value>
				<Type>Boolean</Type>
			</Property>
			<Property>
				<Name>dnaDBurl</Name>
				<Value>jdbc:mysql://*********:3306</Value>
				<Type>String</Type>
			</Property>
			<Property>
				<Name>dnaDBuser</Name>
				<Value>*******</Value>
				<Type>String</Type>
			</Property>
			<Property>
				<Name>dnaDBpwd</Name>
				<Value>*******</Value>
				<Type>String</Type>
			</Property>
		</Properties>
	</Application>
</Root>

By the way I found recorded file from second exception example in the S3 bucket and I didn’t find it in the local folder. So it seems that the file was moved, but it had versioned name like “originalName_0.mp4” as if it was recorded twice. Although it should not be so. Also nothing had been saved in the database.

And when “fileMoverVersionFile” property was set to “false” I could find one file in the S3 and another in the local folder. Both files had the same name, but in the most cases one of them had the size 0 bytes. And there was saved information about this file in the database. The exceptions were the same as now.

I have no idea why it’s happening. Could anybody explain me what I’m doing wrong?

Thanks!

Hi,

I would suggest that you go back to the default mover module and remove everything else.

See if it can be replicated then.

If it can be replicated I would suggest sending in a support request via the Wowza Account Portal. Please include the following information so that we can review your workflow and identify the issue.

[wowza-install-dir]/conf

[wowza-install-dir]/logs

[wowza-install-dir]/transcoder

If you are not sure how to get this information please see this tutorial.

Regards,

Andy_E