Skip to content
Snippets Groups Projects
Commit 67562951 authored by am0ebe's avatar am0ebe
Browse files

update gitignore -> exclude hardware dox and old big dox, but keep todo and scrap/misc notes

parent 1f8021cb
No related branches found
No related tags found
No related merge requests found
*.sublime-workspace *.sublime-workspace
a/hardware
dox/old
File added
File moved
a/setup/beetle.jpg

173 KiB

#!/usr/bin/python3
# more background:
# https://photographylife.com/sensor-crop-factors-and-equivalence
from pprint import pprint as pp
from math import *
sensor_size_to_crop_factor={
#can also calculate it manually by using diagonal sensor size and compare to full frame diagonal sensor size
#https://shuttermuse.com/calculate-cameras-crop-factor/#digital-sensor-sizes-and-their-crop-factor
"full frame":1, #reference: 36mm x 24mm -> diag:43.27
"APS-C (Nikon etc.)":1.5,
"APS-C (Canon)":1.6,
'micro four thirds"':2,
'1"':2.7,
'2/3"':5.5,
'1/1.8"':4.8,
'1/2.3"':5.6,
'1/2.5"':6,
}
focal_len_2_AOV_full_frame={
11:117.1,
14:104.3,
16:96.7,
24:73.7,
35:54.4,
50:39.6,
85:23.9,
100:20.4,
150:13.7,
}
#equivalent_focal_length = focal_length_full_frame * crop_factor
#fov - field of view (= horizontal angle of view)
#size_subject
#distance of subject
def calc_fov(sensor_width_mm,f):
# for sensor_size, crop_factor in sensor_size_to_crop_factor:
# todo
# sensor_size='2/3"'
# crop_factor=sensor_size_to_crop_factor[sensor_size]
# pp(sensor_size)
# pp(crop_factor)
fov=2*atan(sensor_width_mm/(2*f)) * 180/pi #convert to degrees with 180/pi, same as degrees(...)
fov=round(fov,2)
return fov
def calc_distance_to_subject(size_subject_cm, fov):
'''calc distance_to_subject from size_subject and fov using trigonometrics'''
'''depending on lense, the distance_to_subject is fixed or a minimum distance from the img plane'''
# distance_to_subject = (size_subject/2) / tan(radians(fov))
# distance_to_subject = round(distance_to_subject,2)
return distance_to_subject
def main():
size_subject_cm=50
sensor_width_mm=36
f=11
fov=calc_fov(sensor_width_mm,f)
pp(f"fov: {fov}")
d=calc_distance_to_subject(size_subject_cm,fov)
pp(f"distance_to_subject: {d}")
if __name__ == '__main__':
main()
# formulae:
# AOV (in degrees) = 2 ArcTan( sensor width / (2 X focal length)) * (180/π)
# size_subject = 2 * (Tan (AOV/2) X Distance_to_Subject)
a/setup/lab-ground-plan.png

199 KiB

This diff is collapsed.
a/setup/setup-chamber-sketch.png

42.1 KiB

Source diff could not be displayed: it is too large. Options to address this: view the blob.
## software B - calc centroids
################################
g++ -ggdb -o `pkg-config --cflags --libs opencv4`
camera_matrix 5678
distortion_coefficients 76745
where is the data?
PATHIN="/data/ibtrack/move/input/rec$IDSTRING/"
files:
randlist.yml
fileErrorList.yml
## software A - grab pictures from camera
#########################################
Vimba Viewer:
see png to understand components architecture.
test: $INSTALL_DIR/Tools/Viewer/Bin/x86_64bit/VimbaViewer
dox/misc 0 → 100644
## industry cameras
###############################
5gige and 10gige more expensive + might get bandwidthproblems with 6cams? + 10gige need externalframegrabber
usb2 - too old
> usb3
- 4.5m cable
+ usb3 interface card with 4 cams connected
> gige
+ 100m cable
hi res (20MP) and low fps, or
low res (2MP) and high fps
bandwith average. between usb2 and 3
-> can i reduce the fps manually?
multiple cams:
star-network using an ethernet switch (bandwidth will be divided), or
a point-to-point network using an interface card with multiple GigE inputs.
what interface gige card to use?
> needs to support PoE
> ETH from PC provides PoE? if no, then interface card
## Heating:
> cable:
cat6a is better than cat6 or 5e
dont bundle multiple cables together
## lens
420nm and 720nm and 800nm-1300nm IR
IR corrected -> can be used for N IR
problem: spheric abberation = rays farther from center are refracted to different focal point
> most @ max aperture (f1.8)
problem: chromatic aberration = different colors (wavelentgths) have different focal lengths (sa. ~prisma)
> color finging
> longitudinal and lateral CA
solutions:
> achromate = multi-lenses (doublet, triplet).
color correction (blue+red)
> aspheric
different form of lense (not spheric :)
> aspheric achromate
susceptible to scratch.
good quality
-> ir-corrected doublet/triplet
"fast aperture" = max aperture -> eg f1.8 needs to be quick to close all the way
DOF
-> is it possible to use a non-IR gigE cam with a IR-corrected lense, for nightvision?
-> angle + distance
fixed or zoom lens
-> cant get color img _and_ IR img at the same time
-> opt for IR, because it works with light and without
thermal >= nightvision
see through fog/grass? ---
450 2mp cam
600 5mp cam
Lense:
calculate fixed focal length (fest brennweite)
8mm >>
6mm >> 225oi
5mm >> 60cm
<900nm QE ringlichter >> NIR.
calculation:
> 50 cm diameter -> 500mm
> 2.4 MP (1936x1216) -> 1216 pix per line
1216 / 500 = 2.4 px/mm
small insect = 4mm -> 8px -> 8*8 = 64px...
-> can i use SWIR for VS as well? yes, but they are unaffordably expensive
-> SWIR for RaspPi? or usb3? cheaper?
## DOF - Schärfentiefe - Tiefenschärfe
#############################################################
https://photographylife.com/what-is-depth-of-field
app: simple DOF calculator
https://www.edmundoptics.com/c/achromatic-lenses/652/#
> more DOF on
* smaller focal length
* lower aperture eg f16 > landscape
> less DOF
* high aperture eg f1.8
* larger focal length -> zoom 150mm
> BOKEH. nice creamy background
> macro, landscape with FG-POI
> f5.6 sharpest + best middle ground
## crop factor // sensorsize
> equivalent focal length
eg. olympus 2x
8mm > 16mm - narrower
100mm > 200mm -more zoom (fake crop zoom)
> calc equiv. focal length:
https://www.translatorscafe.com/unit-converter/en-US/calculator/lens-efl/
> smaller sensor
= more crop and more "zoom" (narrower field of view, less DOF)
= less light, dynamic range, pixelsize
= less pixels (normally)
> bigger sensor
= less crop, wider field of view -> can crop manually (but more DOF)
##build myself... IoT / embedded camera
raspberryPi + CAM
> 15€ possible?
> https://www.raspberrypi.com/products/raspberry-pi-zero-2-w/
MIPI CSI2 - interface -> all Pis have interface
>> datarates?
>> IR / lowlight compatible?
>> howto make rugged?
nvidia jetson nano?
https://www.nvidia.com/de-de/autonomous-machines/embedded-systems/jetson-nano/
csi(2) - pi: camera serial interface
dsi(2) - pi: display serial interface
csi/dsi -> max len 25cm!
--> via FPD-Link III up to 15m
--> GSML2 ?
###############################
hub/repeater > just cp data to all connected
switch >
router >
codemeter
GenICamBrowser
###############################
## How-To Stream?
SS = Streaming Server
GT 1920:
read stream specs.
search VIMBA ... IP-based.
ndi-python
wrapper for NDI-SDK
ffmpeg as RTMP-SS
nginx
nginx-rtmp-module
linux SS
openCV
VideoCapture( protocol://host:port/script_name?script_params|auth )
Gstreamer
QT
Vimba plugin?
obs-studio/jammy 27.2.3+dfsg1-1 amd64
recorder and streamer for live video content
can use NDI?
vimba: GVSPAdjustPacketSize - find max size automatically
if size < 1500
> enable jumboframes
check:
vimbaviewer > stream > statistics
StatFramesDelivered / StatPacketsReceived != 0
StatFramesDropped == 0
run as root. (or workaround p159 to start app as root and drop rigths immediatly afterwards)
dox/notes 0 → 100644
## SW from tboy old workstation
https://ctrax.sourceforge.net/
walking fly tracker
HOBOWare Lite
temperature logger
StreamPix5
realtime digital video recording to PC memory or hdd
batch processor
https://www.norpix.com/products/streampix/streampix.php
Matlab (.m files)
any2ufmf
fviewflytrax
avidemux
yEd - graph editor - .graphml files
batchprocessor
process sequence files to avi
ssdcleanup
##buy?
https://www.alliedvision.com/en/products/accessories/tripod-adapter/
###############################
## Emilio
###############################
> !! if beetle runs in circle it is in "escape mode" which doesnt represent "natural" behaviour (explore, quick move from a to b)
-> think about different setup. Bigger Arena?
> fps can be low. Try 3!
Depends on movement speed and estimated error
calc error by comparing distance of 40fps with 3fps
> duration of video can be short. eg 10 minutes.
-> no need for many fridges
###############################
## Software A - Frame grab
##############################
# setup Vimba
> see Vimba_installation_under_Linux.pdf
> download and unpack Vimba SDK
> tar -xzf ./Vimba.tgz
> sudo ./VimbaGigETL/Install.sh
## SETUP GT1920C:
> connect lap and cam directly via ethernet
> set MTU to 8228 (jumbo frames/packets)
> set fixed IP for eth-adapter first and then cam (use Vimbaviewer -> force IP)
eth 169.254.100.1
cam 169.254.x.x # on restart will pick random IP...
mac
000f310338D3
000f310338D4
sub 255.255.0.0
(gat 0.0.0.0 local / none)
-> ip address of each adapter needs to be on a unique subnet
-> for multiple cams calc bandwith and use switch
# What bandwidth do i have? Do i need multiple ports?
bandwith = fps * pixel format(bpp) * resolution (* ncams)
StreamBytesPerSecond = 17 * 3 * 1936 * 1216 * 1 = 8 456 448 ~= 8,4 MBps < 125MBps
1. Determine max_speed with highest fps!
2. Take max_speed and reduce fps so it still fits 2*max_speed
-> subsample same video ?
3. calc mean_err through comparing 1. and 2. -> add to 2. as it will be lower.
!! exposure and fps: on 10fps exposure can't be more then 100ms!
!! exposure and fps: on 17fps exposure can't be more then 59ms!
Best practice: set gain to lowest possible val and increase exposure as needed
!! if you use more than one cam on one interface, the available bandwidth has to be shared between interfaces.
###############################
# bandwidth calculations for Alvium G1 240C
MAX BANDWITH for Gigabit ethernet port ~= 125 Mbps
max res: 1936x1216
max fps: 49
pixelformat: RGB8 > different options. eg 10bit/12bit RGB/MONO.
1936 * 1216 * 49 * 1 ~= 115,3 MBps >> MONO8 @ maxFPS
1936 * 1216 * 17 * 1 ~= 40,0 MBps >> MONO8 @ 17FPS
@ 1 Min -> 2.4GB
@ 30 Mins -> 144GB
with 60 recordings: 8.6TB
1936 * 1216 * 17 * 3 ~= 120,1 MBps >> RGB8 @ 17FPS
@ 1 Min -> 7.2GB
@ 30 Mins -> 216.1GB
with 60 recordings: 13 TB
1936 * 1216 * 10 * 3 ~= 70,6 MBps
@ 1 Min -> 4.2 GB
@ 30 Mins -> 127.1 GB
with 60 recordings: 7.6TB
Storage:
ws1:
2TB: 2*1TB (7200RPM)
8Gb DDR3 RAM
Intel Xeon 3.6Ghz (4 cores)
ws2: 4TB: 3 (7200RPM = 6GB/s!) + 1
8GB DDR3 RAM
default bandwidth:115MBps
Save "reasonable" settings in XML.
If dark, set ExposureTimeAbs
Q: how to set fps to 3?
#####################################
## Background Substraction - BGS
####################################
#define bgs_register(x) static BGS_Register<x> register_##x(quote(x))
## >> glue together macro?
quote() adds ""
bgs_register(Tapter)
>> static BGS_Register<Tapter> register_Tapter("Tapter")
/usr/include/opencv4/opencv2/opencv.hpp
PCA?
> difference
virtual();
virtual(){ /*empty*/ }
virtual() = 0; #pure virtual
> also difference
> virtual dtor(): if pointer to base-class deletes object
> pure virtual dtor(): need to also define function body, cuz dtor is special function whis is not overriden
> interface class (needs pure virtual??)
> abc - abstract base class. can't be instantiated anymore
> abc <> interface?
// IplImage is oldskool mat and not supported anymore..
> use smartpointer like so:
auto videoAnalysis = std::make_unique<VideoAnalysis>();
videoCapture = std::make_unique<VideoCapture>();
frameProcessor = std::make_shared<FrameProcessor>();
###########
libarchive stuff
archive_read_xxx()
archive_write_xxx()
struct archive_entry
huge workaround for corrupted files
clock
random
command arg parser
???
#if CV_MAJOR_VERSION > 3 || (CV_MAJOR_VERSION == 3 && CV_SUBMINOR_VERSION >= 9)
IplImage _frame = cvIplImage(img_input);
frame = &_frame;
#else
frame = new IplImage(img_input);
#endif
#smartpointer??
#elif CV_MAJOR_VERSION >= 3
cv::Ptr<cv::BackgroundSubtractorMOG2> mog;
#endif
> What Tboy do... why did he do it?
> forked branch bgslib_qtgui_2.0.0
> Tapter
> adapter for model
> was probably copied from LBAdaptiveSOM.cpp
--> which is disabled in openCV4
Which Background Subtraction Algo to use??
median, mean, framedifference
+ simple, fast,
- not robust if light/bg changes
- slow changes ?
> adaptive bg?
fuzzy?
mixture
NAMES
Kernel?
LBSP?
Multilayer?
Sigma-Delta?
Vibe, T2F ,dp ,lb ,vumeter?
choquet, sugeno, zivkovic, pratimediod, LOBSTER
Test/Use most common >> Ground Truth
Frame Difference
WeightedMovingMean / Variance
LBAdaptiveSOM
MOG2 (Mixture Of Gaussian) MixtureOfGaussianV2.h
KNN (K Nearest Neighbour)
> fast for small fg obj
> TRY!
FuzzySugenoIntegral.h
LSBP - Local Binary Similarity Patterns - (2013)
LSBP-based GSoC ?
SuBSENSE: improved spatiotemporal LBSP + color features (2014)
Combineable with
ROI
Canny Edge Detection
> bg modeling to update BG (eg moving trees) > pixel with threshold
> optic flow (camera is also moving. ) > vectoral estimation of own movement
features
edge
canny edge detector + calc contour
> https://en.wikipedia.org/wiki/Canny_edge_detector
roi
crop
color
RGB - not so robust by itself (sensitive to illumination, shadows, oscillations ...)
YUV
YCrCb - brightness, chroma, color
texture
robust to illumination and shadow
eg Local Binary Pattern (LBP)
https://github.com/murari023/awesome-background-subtraction (2021 new stuff!)
https://learnopencv.com/background-subtraction-with-opencv-and-bgs-libraries/
http://docs.opencv.org/2.4/doc/tutorials/imgproc/gausian_median_blur_bilateral_filter/gausian_median_blur_bilateral_filter.html
https://hackthedeveloper.com/background-subtraction-opencv-python/ #mog2 + knn in python
https://docs.opencv.org/4.3.0/d4/dd5/classcv_1_1bgsegm_1_1BackgroundSubtractorGSOC.html#details #GSOC LSBP ALGO from openCV bgsegm.hpp
https://openaccess.thecvf.com/content_cvpr_workshops_2014/W12/papers/St-Charles_Flexible_Background_Subtraction_2014_CVPR_paper.pdf
https://www.scitepress.org/Papers/2018/66296/66296.pdf #vehicle tracking latvia. 2018, BackgroundSubtractorMOG, BackgroundSubtractorMOG2 (zivkovic)
https://www-sop.inria.fr/members/Francois.Bremond/Postscript/AnhTuanAVSS14.pdf 2014
https://arxiv.org/pdf/1803.07985.pdf # visual animal tracking (2018)
https://arxiv.org/pdf/1507.06821.pdf # Multimodal Deep Learning for Robust RGB-D Object Recognition (2015)
https://towardsdatascience.com/background-removal-with-deep-learning-c4f2104b3157?gi=2ef3a5272e5d (2017 Background removal with deep learning)
https://opencv.org/courses/ #xpensive ai course
https://www.fast.ai/ #free ai course
Build with Python or C++?
##################
# computervision #
##################
background subtraction (bgs)
segmentation
Semantic Segmentation (ai)
detection (feature, object)
classification (category recognition)
Challenges: Occlusion, (Sensor-)Noise, changing external conditions( lighting, Shadows, fog, reflection )
> pre-training if lack of data
> corrupt data to guarantee robust learning
######
# AI #
######
Convolutional Neural Networks (CNNs)
https://www.researchgate.net/publication/337401161_Fundamental_Concepts_of_Convolutional_Neural_Network (2020 37p)
> Kernel
> filter?
> preprocessing > Training > Parameter init > regularization > optimize
RGB-D - RGB + depth >> eg Kinect-Sensors
KNN - k-nearest neighbors algorithm
Monte Carlo tree search algorithm (alphaGo...)
RGB-D descriptor that relies on a K-Means
HMP - hierarchical matching pursuit ( hierarchical sparse-coding method; learn features from multiple channel input)
encoding depth information into three channels (HHA encoding) feature hierachy
cascade of Random Forest classifiers that are fused in a hierarchical manner ?? #Random Forest
metrics:
recall - how many relevant items are retrieved?
precision - how many retrieved items are relevant?
> https://en.wikipedia.org/wiki/Recall_(information_retrieval)
?
automatically learn a fusion strategy for the recognition
task – in contrast to simply training a linear classifier on
top of features extracted from both modalities
MISRE - Multiple Input Structures with Robust Estimator
gradient-based learning
> MNIST supervised image classification
MNIST - large db of hand-written digits (used for ML and ImagePRocessing)
> also see classifiers: https://en.wikipedia.org/wiki/MNIST_database#Classifiers
multimodal learning
the parameters of all layers were adapted using a fixed learning rate schedule
At test time the task of the CNN is to assign the correct class label to a previously unseen object instance.
Tasks:
Classification, clustering, regression, Summarization, Categorization, Natural language processing,
##################################
# STAT
##################################
Regression
Hadamard Product
Bernoulli Distribution
Discrete Uniform Distribution
##################################
# CMAKE
##################################
sublime: install package CMAKE (syntax hilight)
apt install libarchive-dev python3-pybind11
##################################
init(img_input, img_output, img_bgmodel);
#if CV_MAJOR_VERSION > 3 || (CV_MAJOR_VERSION == 3 && CV_SUBMINOR_VERSION >= 9)
IplImage _frame = cvIplImage(img_input);
frame = &_frame;
#else
frame = new IplImage(img_input);
#endif
point2f(y,x) # inverted!
BGR #not rgb ...
Scalar = 4 element vector
CV_RGB(r, g, b) #from bgs
Types:
#depth
CV_8U unsigned char
CV_8S char
CV_16U unsigned short
CV_16S short
CV_32S int
CV_32F float
CV_64F double
# + channels C1..4 eg RGBA:
# eg Mat M(2,2, CV_8UC3, Scalar(0,0,255));
0 to 255 for CV_8U images
0 to 65535 for CV_16U images
0 to 1 for CV_32F images
# Camera Calibration:
########################
distortion_coefficients - 5 values
camera_matrix - 3x3 matrix - eg focal length etc.
different methods:
https://docs.opencv.org/4.x/d7/d21/tutorial_interactive_calibration.html
https://docs.opencv.org/4.x/d6/d55/tutorial_table_of_content_calib3d.html
# using chess board, to get values
https://github.com/abidrahmank/OpenCV2-Python-Tutorials/blob/master/source/py_tutorials/py_calib3d/py_calibration/py_calibration.rs
# Matrices
########################
Fill matrix with random values
Mat R = Mat(3, 2, CV_8UC3);
randu(R, Scalar::all(0), Scalar::all(255));
Formatted output
cout << "R (csv) = " << endl << format(R, Formatter::FMT_CSV ) << endl << endl;
#sa FMT_C/PYTHON/NUMPY/CSV
InputArray = vector or Mat
type()
create(r,c,t)
copyTo()
clone() for deep copy
cv::Mat F = cv::Mat::ones(3, 4, CV_32FC1) * 3; //all 3's
traverse: https://programmersought.com/article/72762264465/
##roi - region of interest
x, y, w, h
// roi as subset of full data image
Mat D (A, Rect(10, 10, 100, 100) ); // using a rectangle
Mat E = A(Range::all(), Range(1,3)); // using row and column boundaries
locateROI, adjustROI,
# CommandLineParser
@ -> positional arg
no default value -> has_XX
else: get<T>("XX")
<none> dflt value to ensure strings arent ""
# MISC
########################
glob( dir, file )
randu() # https://docs.opencv.org/4.x/d2/de8/group__core__array.html#ga1ba1026dca0807b27057ba6a49d258c0
cvtColor(img, img, COLOR_BGR2Luv); #convert color
sounds interesting:
selectROI, selectROIs, reduce, qt, dilate, erode,
convexhull, findcontours, checkChessboard, calibrateCamera
hough ?
hu-invariants, centroid
moments (weigthed avg of pixel intensities)
spatial <> central <> central normalized
randPattern
BS:
https://docs.opencv.org/4.x/da/d5c/tutorial_canny_detector.html
https://docs.opencv.org/4.x/d1/dc5/tutorial_background_subtraction.html
https://web.archive.org/web/20140418093037/http://bmc.univ-bpclermont.fr/
https://web.archive.org/web/20140221195750/http://docs.opencv.org/trunk/doc/tutorials/video/background_subtraction/background_subtraction.html
https://www.pyimagesearch.com/2020/07/27/opencv-grabcut-foreground-segmentation-and-extraction/
https://learnopencv.com/applications-of-foreground-background-separation-with-semantic-segmentation/
https://docs.opencv.org/4.x/d2/d55/group__bgsegm.html
https://docs.opencv.org/4.x/d5/de8/samples_2cpp_2segment_objects_8cpp-example.html#_a15
tuts:))
https://docs.opencv.org/4.x/d6/d00/tutorial_py_root.html
https://datahacker.rs/opencv-thresholding/
# median frame as bg + frame difference
https://learnopencv.com/simple-background-estimation-in-videos-using-opencv-c-python/
https://learnopencv.com/contour-detection-using-opencv-python-c/
https://learnopencv.com/deep-learning-with-opencvs-dnn-module-a-definitive-guide/
moments
https://docs.opencv.org/4.x/d0/d49/tutorial_moments.html
#BGS
https://docs.opencv.org/4.x/d8/d38/tutorial_bgsegm_bg_subtraction.html
# alliedvision-vimba-sdk
#######################
AsynchronousGrab example > to use with opencv
/tmp/mozilla_sugu0/Vimba_installation_under_Linux-1.pdf
C-api
https://www.ansatt.hig.no/ares/Support/Camera%20drivers/AVT/Vimba/1.1/VimbaC/Documentation/Vimba%20C%20Manual.pdf
...@@ -193,8 +193,39 @@ X) HPC - High Performance Cluster ...@@ -193,8 +193,39 @@ X) HPC - High Performance Cluster
Z) laptop/fon maintenance Z) laptop/fon maintenance
############################### ###############################
> sync+manage bookmarks of different FIREFOXES. Delete one.
> add symbols to 'cmds' for quick copy-paste
> sub: alt+5 can't toggle to 5th pane.
shortcut for toggling panes via number and/or left-right
shortcut for replace next in an intuitive manner (no mouse!)
## NOTES
###############################
"i conduct experiments in a lab, that i build up in a prestigious scientific institute"
"as a craftsman, i hone my skill to build/create things"
"i use my skills to earn €"
"i live and work with integrity"
-------------------------------
4-5h/d @ 4-5/w -> finetune DW/SW
get feedback (occassionaly)
milestone 1: build up hardware
> fridges,cams,light,arena,workstation,
milestone 2: get videodata
> 2.1 via vv
> 2.2 via own SW!
milestone 3: collect all data using 1 camunit
milestone 4: collect all data using 6 camunits
milestone 5: conduct experiments
milestone 6: analyze I - get simple vars
movement: calc coordinates (roi,bgs substraction,bgs midpoint,..)
aggregate CSV (coord + time + temp/rH)
milestone 7: analyze II - get more vars
> calc speed,angles,distance
> 5 & 6 could be swapped!
Done Done
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment